This  investigative  study  demonstrates  the  benefits  of  addressing  human 
considerations  early  in  the  system  development  life  cycle  that  will  bring  long¬ 
term  benefit  to  program  managers  and  systems  engineers.  The  approach  used 
a  retrospective  content  analysis  of  documents  from  weapon  systems  acquisi¬ 
tion  programs,  namely  Major  Defense  Acquisition  Programs.  Binary  logistic 
regression  analyses  were  conducted  to  predict  the  effect  of  the  presence  of 
words  relating  to  Human  Systems  Integration  on  the  success  of  programs. 


This  investigative  study  corroborates  the  idea  that  some  benefit  may 
be  derived  from  implementing  Human  Systems  Integration  during  the 
weapon  systems  acquisition  life  cycle. 
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In  1981,  U.S.  congressional  watchdogs  recommended  improving  weapon 
systems  design  by  addressing  human  consideration  problems  early  during 
system  acquisition  (U.S.  General  Accounting  Office,  1981).  Today,  the  pro¬ 
cess  by  which  human  considerations  are  included  in  the  planning  and 
implementation  of  a  system  is  known  as  Human  Systems  Integration  (HSI) 
(International  Council  on  Systems  Engineering  [INCOSE],  2012).  The  ben¬ 
efit  of  including  HSI  in  weapon  systems  design  and  acquisition  is  best 
realized  by  giving  HSI  early  attention  and  priority  during  the  planning  stage. 
This  article  will  demonstrate  that  a  decreased  percentage  of  HSI  words  in 
documents  originated  during  acquisition  will  coincide  with  unforeseen 
costs,  delays,  and  performance  problems. 


The  benefit  of  including  HSI  in  weapon  systems  design 
and  acquisition  is  best  realized  by  giving  HSI  early 
attention  and  priority  during  the  planning  stage. 


In  fact,  HSI-related  content  in  acquisition  documents  may  influence  pro¬ 
gram  success.  Typically,  a  program  is  considered  successful  if  it  avoids 
cost  overruns,  avoids  performance  breaches,  or  avoids  schedule  breaches. 
Systems  Engineering  (SE)  is  the  interdisciplinary  approach  for  develop¬ 
ing  systems  (INCOSE,  2012).  HSI  is  an  important  part  of  SE,  and  thus  the 
acquisition  life  cycle  (Karwowski,  2012).  Understandably,  the  decisions  that 
program  managers  and  systems  engineers  make  early  in  the  acquisition  life 
cycle  will  affect  program  success  and  life-cycle  costs.  For  example,  to  help 
organizations  with  incorporating  HSI  into  their  design  process,  Handley 
and  Knapp  (2014)  have  created  the  Human  Viewpoint  tool  for  early  imple¬ 
mentation  of  HSI  into  the  acquisition  life  cycle.  Ahram  and  Karwowski 
(2012)  warn  that  failing  to  address  costs  related  to  human  performance  (e.g., 
Human  Total  Ownership  Cost)  early  in  the  life  cycle  will  lead  to  schedule 
overruns,  diminished  system  performance,  inadequate  training,  and  mis¬ 
aligned  plans  for  manpower  and  personnel  allocation.  Cramer,  Sudhoff,  and 
Zivi  (2011)  posit  that  integrating  survivability  as  a  design  objective  early  in 
the  life  cycle  can  benefit  the  design  process.  Assessing  human  capabilities 
during  technology  readiness  level  evaluation,  according  to  Wallace,  Bost, 
Thurber,  and  Hamburger  (2007),  can  help  the  program  avoid  cost  overruns. 
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The  interaction  between  humans  and  the  systems  they  use  affects  program 
success,  as  well  as  life-cycle  costs.  The  documents  from  weapon  systems 
acquisition  programs,  namely  major  defense  acquisition  programs  (MDAP), 
contain  a  history  of  each  program’s  system  development  life  cycle,  and  this 
history  indicates  what  considerations  were  involved  in  the  system  devel¬ 
opment  life  cycle.  It  follows  then  that  HSI-related  content  in  acquisition 
documents  is  interrelated  with  program  success. 

This  article  is  essentially  an  investigative  study  or  retrospective  content 
analysis  of  MDAP  documents.  The  author’s  goal  is  to  present  a  sound  argu¬ 
ment  that  omitting  HSI  during  weapon  systems  acquisition  will  coincide 
with  acquisition  life-cycle  cost  overruns,  as  well  as  schedule  slippages  and 
performance  breaches.  More  specifically,  this  investigative  study  addresses 
the  gap  in  knowledge  among  weapon  systems  acquisition,  HSI,  and  acquisi¬ 
tion  life-cycle  cost,  performance,  and  schedule  (Figure  1). 


FIGURE  1.  VENN  DIAGRAM  OF  INTERRELATIONSHIPS  AMONG 
WEAPON  SYSTEMS,  HSI,  AND  LIFE-CYCLE  COST, 
PERFORMANCE,  AND  SCHEDULE 


Performance,  and 
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Problem  Statement 

Stakeholders,  such  as  program  managers  and  systems  engineers,  strive 
to  mitigate  unforeseen  costs  during  the  system  development  life  cycle.  One 
way  to  achieve  this  objective  is  to  prioritize  human  considerations  early  in 
the  system  development  life  cycle.  A  program  manager  or  systems  engineer 
could  predict  program  success  using  regression  analysis  of  historical  data. 
Program  documents,  such  as  Selected  Acquisition  Reports  (SARs),  provide  a 
valuable  source  of  historical  data  about  weapon  systems  programs  (Assidmi, 
Sarkani,  &  Mazzuchi,  2012;  Bielecki  &  White,  2005;  Birchler,  Christie,  & 
Groo,  2011).  For  purposes  of  this  investigative  study,  logistic  regression  and 
HSI-related  terminology  in  documents  will  be  used  to  make  predictions 
about  program  success. 

Presence  of  HSI-related  terminology  is  defined  by  the  percentage  of  HSI- 
related  words  per  document.  Program  success  is  defined  by  avoidance  of  cost 
overrun,  avoidance  of  performance  breach,  or  avoidance  of  schedule  breach. 
Overall,  the  objective  is  to  conduct  a  retrospective  content  analysis  of  MDAP 
documents  as  an  approach  by  which  to  seek  the  presence  of  HSI-related 
terminology  in  weapon  systems  acquisition.  This  approach  is  designed  to 
demonstrate  that  an  earlier  presence  of  HSI-related  terminology  will  pre¬ 
dict  better  outcomes  for  weapon  systems  acquisition  programs  with  money 
saved,  time  saved,  and  good  performance.  A  definitive  research  question 
that  addresses  the  problem  identified  by  this  investigative  study  follows. 

Does  the  percentage  ofHSI  words  within  the  document  coin¬ 
cide  with  unforeseen  cost  overruns,  performance  breaches, 
and  schedule  slippages? 

Although  the  findings  of  this  investigative  study  did  not  yield  strongly 
predictive  regression  models,  significant  findings  emerged  that  suggest 
schedule  slippages  and  cost  overruns  may  be  associated  with  a  reduction 
of  HSI-related  terminology.  From  the  findings,  the  presence  of  terminology 
about  human  factors  engineering,  habitability,  and  survivability  as  well 
as  manpower,  personnel,  and  training  suggests  that  a  program  will  likely 
succeed.  This  finding  corroborates  the  idea  that  a  solution  to  the  problem— 
specifically  saving  time,  saving  money,  and  improving  performance— will 
be  the  inclusion  of  HSI-related  content  early  in  the  weapon  systems  acqui¬ 
sition  life  cycle. 
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Data  Collection 

The  data  collection  effort  identified  HSI-related  terminology  in  each 
document,  including  HSI  word  percentages.  Program  success  data  were 
collected  with  regard  to  cost,  schedule,  and  performance.  Weapon  systems 
acquisition  programs,  specifically  MDAPs,  were  also  identified.  Additional 
data  were  collected  to  identify  when  each  document  was  published  with 
regard  to  its  corresponding  program’s  Milestone  B. 

HSI-Related  Terminology 

As  shown  in  Table  1,  the  HSI-related  terminology  in  this  investigative 
study  consisted  of  words  that  refer  to  the  nine  HSI  domains  defined  by 
the  Department  of  the  Air  Force  (2014).  Also  included  were  the  terms  HSI 
and  MANPRINT,  which  are  synonymous  (Drillings,  2014).  It  is  helpful 
to  note  that  HSI  is  defined  differently  among  organizations.  For  example. 
Headquarters  Department  of  the  Army  (2014)  defined  seven  HSI  domains: 


1.  manpower 

2.  personnel  capabilities 

3.  training 

4.  human  factors  engineering 

5.  system  safety 

6.  health  hazards 

7.  soldier  survivability 


Department  of  the  Navy  (2009)  defined  seven  slightly 
different  HSI  domains: 


Department  of  the  Air  Force  (2014),  however,  defined  nine 
versus  seven  HSI  domains: 


1.  manpower 

2.  personnel 

3.  training 

4.  environment 


1. 

2. 

3. 

4. 

5. 

6. 
7. 


manpower 
personnel 
training 

human  factors  engineering 
environmental  safety  and  occupational  health 
habitability 
personnel  survivability 
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5.  safety 

6.  occupational  health 

7.  human  factors  engineering 

8.  survivability 

9.  habitability 


This  investigative  study  refers  to  the  most  complete  list  of  HSI  domains,  as 
identified  by  Department  of  the  Air  Force  (Table  1). 


TABLE  1.  HSI-RELATED  WORDS  AND  CORRESPONDING  HSI  DOMAINS 

HSI  Domain 

HSI-Related  Words 

NA 

HSI,  Manpower  and  Personnel  Integration 
(MANPRINT) 

Human  Factors  Engineering 

fatigue,  human,  people,  perform, 
performance,  performed,  performing, 
performs,  situational  awareness,  troops, 
usability,  utility,  workload 

Habitability 

habitability,  shelter 

Survivability 

survivability 

Environment 

environment,  environmental 

Safety 

protect,  protected,  protection,  protective, 
safety,  secure,  security 

Occupational  Health 

health 

Manpower 

interoperability,  maintain,  maintainability, 
maintained,  maintained  maintaining, 
maintains,  maintenance,  manpower,  manned, 
manning,  operability,  operate,  operated, 
operates,  operating,  operational 

Personnel 

infantry,  manage,  managed,  management, 
manager,  manages,  managing,  personnel, 
pilot,  role,  staff,  warfighter,  warrior 

Training 

instructor,  train,  trained,  training 

To  indicate  HSI-related  terminology  within  each  program  document,  data 
consisted  of  word  percentages  for  each  HSI-related  word  of  interest  for  each 
of 546  program  documents.  These  word  percentages  were  calculated  using 
word  counts  for  each  HSI-related  word  of  interest  and  the  total  word  count 
for  each  document.  Table  2  shows  the  number  of  documents  per  program 
and  the  range  of  word  counts  per  program. 
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I  TABLE  2.  MAJOR  DEFENSE  ACQUISITION  PROGRAMS  I 

Program  Name 

Total 

Documents 

Range  of 
Word  Counts 

Advanced  Deployable  System 

7 

504  to  1,237 

DDG  1000  Zumwalt  Class  Destroyer 

37 

107  to  17,317 

F-22  Raptor  Advanced  Tactical  Fighter 
Aircraft 

71 

86  to  9,378 

F-35  Joint  Strike  Fighter  Aircraft 

58 

68  to  17,700 

Ground/Air  Task-Oriented  Radar 

6 

228  to  3,807 

EA-18G  Growler  Aircraft 

35 

89  to  39,656 

F/A-18E/F  Super  Hornet 

12 

801  to  2,574 

Joint  Land  Attack  Cruise  Missile  Defense 
(JLACMD)  Elevated  Netted  Sensor  System 

16 

161  to  31,314 

Joint  Tactical  Radio  System 

45 

275  to  5,197 

Littoral  Combat  Ship 

27 

73  to  64,661 

Mine  Resistant  Ambush  Protected  Vehicle 

48 

115  to  51,790 

P-8A  Poseidon  Multi-Mission  Maritime 
Aircraft 

22 

178  to  31,658 

Space  Based  Infrared  System  High 

48 

194  to  4,390 

V-22  Osprey  Joint  Services  Advanced 
Vertical  Lift  Aircraft 

47 

117  to  9,713 

Vertical  Take-off  and  Landing  Tactical 
Unmanned  Aerial  Vehicle 

16 

150  to  2,721 

Warfighter  Information  Network-Tactical 

51 

228  to  31,132 

Data  were  collected  from  the  546  documents  and  entered  into  SPSS 
Statistics  Version  22.0  for  Windows.  HSI  words  within  the  sampled  doc¬ 
uments  ranged  from  zero  to  2,262.  The  average  number  of  HSI  words  was 
42.60  ( SD  =  160.73).  Total  words  for  the  sampled  documents  ranged  from  68 
to  64,661,  and  the  total  word  count  was  2,010.53  on  average  (SD  =  5,109.40). 

HSI-related  words  were  separated  into  three  categories  (Table  3).  Some 
overlap  occurred  among  each  of  the  HSI  domains,  and  some  words  fit  into 
more  than  one  domain  description.  Typically,  environmental,  safety,  and 
occupational  health  issues  are  grouped  together  and  identified  with  the 
acronym  ESOH,  as  are  manpower,  personnel,  and  training  issues,  which 
are  identified  as  MPT.  For  this  investigative  study,  these  same  groups 
were  identified,  thus  the  data  included  ESOH  and  MPT  word  percentages. 
Because  the  terms  habitability  and  survivability  generated  a  small  quantity 
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of  words,  they  were  grouped  together  with  Human  Factors  Engineering 
(HFE),  along  with  the  terms  HSI  and  MANPRINT,  thus  the  data  included 
HFE/Hab/Surv  word  percentages. 


TABLE  3.  HSI-RELATED  WORDS  AND  CORRESPONDING 

HSI  CATEGORIES 

HSI  Category 

HSI-Related  Words 

HFE/Hab/Surv 

HSI,  MANPRINT,  fatigue,  habitability,  human,  people, 
perform,  performance,  performed,  performing, 
performs,  shelter,  situational  awareness,  survivability, 
troops,  usability,  utility,  workload 

ESOH 

environment,  environmental,  health,  protect,  protected, 
protection,  protective,  safety,  secure,  security 

MPT 

infantry,  instructor,  interoperability,  maintain, 
maintainability,  maintained,  maintained  maintaining, 
maintains,  maintenance,  manage,  managed, 
management,  manager,  manages,  managing,  manpower, 
manned,  manning,  operability,  operate,  operated, 
operates,  operating,  operational,  personnel,  pilot,  role, 
staff,  train,  trained,  training,  warfighter,  warrior 

Success  Metrics 

Program  success  data  were  collected  to  investigate  each  MDAP’s  cost 
overruns,  schedule  slippages,  and  performance  breaches.  These  success 
metrics  were  (a)  SAR-identified  cost  breaches,  (b)  SAR- identified  schedule 
breaches,  (c)  SAR-identified  performance  breaches,  (d)  GAO  assessment 
indicating  total  program  over  budget,  (e)  GAO  assessment  indicating  pro¬ 
gram  unit  cost  increase,  and  (f)  Weapon  Book  program  amount  spent  went 
over  budget.  None  of  these  six  metrics  had  absolutely  complete  data  because 
data  were  not  available  for  each  program  for  each  fiscal  year.  To  collect 
ample  data,  all  six  metrics  were  considered  for  this  investigative  study. 

MDAPs 

Several  factors  were  considered  in  the  selection  of  the  sample  of  MDAPs 
analyzed  in  this  study.  Each  program  needed  sufficient  documentation  for 
the  HSI  word  analysis  as  well  as  cost,  schedule,  and  performance  data.  Table 
4  lists  the  16  MDAPs  alongside  their  common  names.  MDAP  documents 
were  collected  between  June  2013  and  October  2014,  primarily  from  the 
Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  data¬ 
base  (DAMIR,  n.d.).  Additional  MDAP  documents  were  collected  from  the 
Acquisition  Decision  Memoranda  (ADM)  Web  site  (ACQWeb)  (Acquisition 
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Decision  Memoranda,  2014).  The  objective  was  to  acquire  acquisition  pro¬ 
gram  documents  that  are  consistent  from  program  to  program;  the  DAMIR 
database  and  ADM  Web  site  made  this  possible. 


TABLE  4.  HSI-RELATED  WORDS  AND  CORRESPONDING  HSI  DOMAINS  1 

Program  Name 

Common  Name 

Advanced  Deployable  System 

ADS 

DDG  1000  Zumwalt  Class  Destroyer 

DDG  1000 

F-22  Raptor  Advanced  Tactical  Fighter  Aircraft 

F-22 

F-35  Joint  Strike  Fighter  Aircraft 

JSF 

Ground/Air  Task  Oriented  Radar 

G/ATOR 

EA-18G  Growler  Aircraft 

EA-18G 

F/A-18E/F  Super  Hornet 

F/A-18E/F 

Joint  Land  Attack  Cruise  Missile  Defense  (JLACMD) 
Elevated  Netted  Sensor  System 

JLENS 

Joint  Tactical  Radio  System 

JTRS 

Littoral  Combat  Ship 

LCS 

Mine  Resistant  Ambush  Protected  Vehicle 

MRAP 

P-8A  Poseidon  Multi-Mission  Maritime  Aircraft 

P-8A 

Space  Based  Infrared  System  High 

SBIRS  High 

V- 22  Osprey  Joint  Services  Advanced  Vertical  Lift 
Aircraft 

V-22 

Vertical  Take-off  and  Landing  Tactical  Unmanned  Aerial 
Vehicle 

VTUAV  Fire 

Scout 

Warfighter  Information  Network-Tactical 

WIN-T 

Note.  If  a  program  had  any  increments,  then  the  increment  was  noted,  and  appropriate 
data  for  that  increment  were  collected. 

None  of  the  MDAP  documents  identifying  HSI-related  terminology  were 
used  to  obtain  cost,  schedule,  or  performance  data.  In  addition  to  collecting 
MDAP  documents,  other  sources  of  data  were  collected  to  identify  program 
cost  overruns,  schedule  slippages,  and  performance  breaches  (Table  5). 
Furthermore,  none  of  the  documents  that  were  used  for  obtaining  infor¬ 
mation  about  program  cost  overruns,  schedule  slippages,  and  performance 
breaches  were  used  to  obtain  HSI  data. 
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I  TABLE  5.  BREACH  DATA  SOURCES  j 

Breach  Data  Source  and  Description 

Types  of 
Breaches 

Origin 

Selected  Acquisition  Reports  (SAR) 
indicate  whether  or  not  a  program  has 
experienced  cost  breach,  schedule 
breach,  or  performance  breach.  Data 
indicating  breaches  were  assigned  Is 
for  breach  occurrence  and  Os  for  no 
breach. 

cost  overruns, 
schedule 
slippages,  and 
performance 
breaches 

DAMIR  database 

(DAMIR,  n.d.) 

Government  Accountability  Office 
(GAO)  assessments  for  major 
weapon  programs  report  whether  a 
program  has  gone  over  budget  and 
whether  a  program’s  unit  cost  has 
increased.  Data  indicating  that  the 
total  program  had  gone  over  budget 
were  assigned  Is,  as  well  as  data 
indicating  that  the  program’s  unit 
cost  had  increased;  absences  of  these 
conditions  were  assigned  Os. 

cost  overruns 

GAO  Web  site 

(GAO,  2014) 

The  Comptroller’s  Weapon  Books 
report  the  Budget  Request  amounts 
that  a  program  requested  for  a 
given  fiscal  year  and  later  what  the 
program  actually  spent  during  that 
fiscal  year.  A  cost  overrun  can  be 
determined  from  comparing  what 
amount  was  requested  and  then  what 
amount  was  actually  spent.  Is  were 
assigned  to  indicate  that  a  program 
went  over  budget  for  the  given  fiscal 
year,  and  Os  were  assigned  in  the 
absence  of  this  condition. 

cost  overruns 

Comptroller’s 

Web  site 

(Under  Secretary 
of  Defense 
Comptroller,  2014) 

Experimental  Design 

Because  the  regression  analysis  was  intended  to  measure  six  dependent 
variables,  six  analyses  were  conducted  for  each  of  the  six  dependent  vari¬ 
ables:  (a)  SAR  cost  breach,  (b)  SAR  performance  breach,  (c)  SAR  schedule 
breach,  (d)  GAO  total  program  over  budget,  (e)  GAO  program  unit  cost 
increase,  and  (f)  Weapon  Book  amount  spent  over  budget.  Subcategories 
of  HSI  word  percentage  per  MDAP  document  included  (a)  ESOH  word  per¬ 
centage,  (b)  HFE/Hab/Surv  word  percentage,  and  (c)  MPT  word  percentage. 
The  independent  variables  were  the  subcategories  of  HSI  word  percentage 
per  MDAP  document. 
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Data  Analysis 

As  shown  in  Table  5,  qualitative  breach  data  were  categorized  with 
Os  and  Is.  When  dependent  variables  are  qualitative,  a  logistic  regression 
equation  can  be  used  to  create  a  model  of  the  probability  that  the  dependent 
variable’s  value  will  be  either  0  or  1  (Chatterjee  &  Hadi,  2012).  After  the 
regression  model  has  been  created,  the  data  are  compared  to  the  model  to 
discover  what  ratio  of  the  data  was  classified  0  or  1  correctly  (Chatterjee  & 
Hadi,  2012).  Binary  logistic  regression,  the  data  analysis  method  selected 
for  this  data  set’s  interpretation,  is  a  method  for  modeling  probabilities  when 
the  outcome  falls  between  0  and  1. 


Because  each  budget  for  each  MDAP  is  unique  from  the  budgets  of  other 
MDAPs,  the  numeric  values  for  each  budget  would  consist  of  different 
numbers  that  cannot  be  directly  compared.  To  assess  the  data,  one  may  ask 
the  question,  “Did  an  MDAP  go  over  budget,  or  didn’t 
it?”  Answering  this  question  opens  the  possibility 
to  make  the  data  dichotomous  with  Os  for  no 
and  Is  for  yes.  These  dichotomous  data  are 
qualitative  because  they  yield  a  qualitative 
value,  such  as  good  or  bad  (Chatterjee  & 

Hadi,  2012).  In  this  manner,  data  regard¬ 
ing  whether  or  not  there  was  a  breach 
can  be  assigned  (categorized)  Os  and  Is, 
and  the  appropriate  statistical  method  is 
binary  logistic  regression. 

Shown  here  is  the  model  for  the  predic¬ 
tive  formulae  that  were  the  outcome  of  this 
study’s  analysis.  Because  predictive  formulae 
from  logistic  regression  are  nonlinear,  they  must  be 
transformed.  The  probability  of  something  happening,  such  as  a  cost  over¬ 
run,  is  the  odds  ratio.  The  logit  is  identified  by  finding  the  logarithm  of  the 
odds  ratio.  This  transformation  ensures  that  the  predictive  formulae  will 
be  linear  (Chatterjee  &  Hadi,  2012). 


logit[Pr(Y=Unforeseen  cost  overrun,  performance 
breach,  or  schedule  slippage)]  =  BQ  +/-  B^ESOH  word 
percentage)  +/-  B2(HFE/Hab/Surv  word  percentage) 
+/-  B3(MPT  word  percentage) 
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Results 

To  examine  the  hypothesis,  a  series  of  binary  logistic  regressions  was 
conducted.  Due  to  the  exploratory  nature  of  this  investigative  study,  an 
alpha  level  of  0.10  was  employed  for  assessment  of  statistical  significance. 
The  hypothesis  is  shown  here. 

The  percentage  ofHSI  words  within  the  document  will  coin¬ 
cide  with  unforeseen  cost  overruns,  performance  breaches, 
and  schedule  slippages. 

Results  of  the  six  regression  analyses  are  presented  in  Table  6,  while  regres¬ 
sion  coefficients  are  presented  in  Table  7.  Four  of  the  six  regressions  yielded 
significant  models.  SAR  schedule  breach,  GAO  total  program  over  budget, 
GAO  program  unit  cost  increase,  and  Weapon  Book  amount  spent  over 
budget  were  associated  with  either  ESOH  word  percentage,  HFE/Hab/Surv 
word  percentage,  or  MPT  word  percentage.  These  four  regressions  were 
further  analyzed  to  assess  the  hypothesis. 
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TABLE  6.  MODEL  FIT  FOR  SIX  BINARY  LOGISTIC 

REGRESSION  ANALYSES 

Dependent  Variable 

x2(  7) 

P 

Nagelkerke  R 

1.  SAR  cost  breach 

9.15 

.242 

.03 

2.  SAR  performance  breach 

5.90 

.552 

.07 

3.  SAR  schedule  breach* 

20.21 

.005 

.06 

4.  GAO  total  program  over  budget* 

28.00 

<  .001 

.13 

5.  GAO  program  unit  cost  increase* 

22.29 

.002 

.12 

6.  Weapon  Book  amount  spent  over 
budget  * 

23.83 

.001 

.14 

Note.  An  asterisk  (*)  indicates  significance  at  the  p  < 

.10  level. 

TABLE  7.  MODEL  COEFFICIENT  DETAILS  FOR 
BINARY  LOGISTIC  REGRESSIONS 


Regression  B  SE  Wald  p  O.R. 

SAR  schedule  breach  + 

MPT  word  -0.74  0.34  4.75  .029  0.48 

percentage 

GAO  total  program  over  budget + 

MPT  word  -0.98  0.59  2.78  .095  0.38 

percentage 

GAO  program  unit  cost  increase  f 


MPT  word 
percentage 

-1.05 

0.51 

4.23 

.040 

0.35 

Weapon  Book  amount  spent  over  budget + 

ESOH  word 
percentage 

5.23 

2.86 

3.35 

.067 

187.23 

HFE/Hab/Surv  word 
percentage 

-4.90 

1.77 

7.64 

.006 

0.01 

MPT  word 
percentage 

1.06 

0.64 

2.74 

.098 

2.87 

Note.  A  dagger  (+)  indicates  model  significance  at  the  p  < 

.10  level. 

To  examine  the  hypothesis,  the  three  predictors  (ESOH  word  percentage, 
HFE/Hab/Surv  word  percentage,  and  MPT  word  percentage)  of  the  four  sig¬ 
nificant  models  were  examined  (Table  6).  Model  coefficients  are  presented 
in  Table  7.  For  SAR  schedule  breach,  MPT  word  percentage  was  a  significant 
predictor  ( B  =  -0.74,  p  =  .029,  OR  =  0.48),  suggesting  that  as  the  percentage  of 
MPT  words  increased,  the  likelihood  of  a  SAR  schedule  breach  decreased. 


Defense ARJ,  January  2016,  Vol.  23  No.  1 : 78-101 


91 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


For  GAO  total  program  over  budget,  MPT  word  percentage  was  a  significant 
predictor  ( B  =  -0.98,  p  =  .095,  OR  =  0.38),  suggesting  that  as  the  percentage 
of  MPT  words  increased,  the  likelihood  of  a  GAO  total  program  over  budget 
decreased.  For  GAO  program  unit  cost  increase,  MPT  word  percentage  was 
a  significant  predictor  ( B  =  -1.05,  p  =  .040,  OR  =  0.35),  suggesting  that  as  the 
percentage  of  MPT  words  increased,  the  likelihood  decreased  for  a  GAO 
program  unit  cost  increase.  For  Weapon  Book  amount  spent  over  budget, 
ESOH  word  percentage  was  a  significant  predictor  ( B  =  5.23,  p  =  .067,  OR 
=  187.23),  suggesting  that  as  the  percentage  of  ESOH  words  increased,  the 
likelihood  of  a  Weapon  Book  amount  spent  over  budget  increased.  Also  for 
Weapon  Book  amount  spent  over  budget,  HFE/Hab/Surv  word  percent¬ 
age  was  a  significant  predictor  ( B  =  -4.90,  p  =  .006,  OR  =  0.01),  suggesting 
that  as  the  percentage  of  HFE/Hab/Surv  words  increased,  the  likelihood 
of  a  Weapon  Book  amount  spent  over  budget  decreased.  Last,  MPT  word 
percentage  was  a  significant  predictor  of  Weapon  Book  amount  spent  over 
budget  ( B  =  1.06,  p  =  .098,  OR  =  2.87),  suggesting  that  as  the  percentage  of 
MPT  words  increased,  the  likelihood  of  a  Weapon  Book  amount  spent  over 
budget  also  increased. 

Predictive  Equations 

For  each  of  the  four  significant  models,  each  regression  was  solved  to 
provide  a  predictive  formula  for  the  relationship  between  HSI  word  per¬ 
centage  outcomes  and  the  variables  of  interest  to  Hypothesis  One.  Each  of 
these  predictive  formulae  consider  ESOH  word  percentage,  HFE/Hab/Surv 
word  percentage,  or  MPT  word  percentage  as  subcategories  of  HSI  word 
percentage.  The  first  significantly  predictive  model  suggested  that  MPT 
word  percentage  was  the  only  factor  that  made  a  unique  contribution  to  the 
prediction  of  SAR  schedule  breaches.  Increased  percentage  of  MPT  words 
contributed  to  a  lower  likelihood  of  SAR  schedule  breaches.  This  model 
resulted  in  the  final  equation  shown  here. 

logit[Pr(Y=SAR  schedule  breach)]  =  0.06  -  0.74(MPT 

word  percentage) 

The  second  significantly  predictive  model  suggested  that  MPT  word  per¬ 
centage  was  again  the  only  factor  that  made  a  unique  contribution  to  the 
prediction  of  GAO  total  program  over  budget  cost  overruns.  Increased 
percentage  of  MPT  words  contributed  to  a  lower  likelihood  of  GAO  total 
program  over  budget  cost  overruns.  This  model  resulted  in  the  final  equa¬ 
tion  shown  here. 
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logit[Pr(Y=GAO  total  program  over  budget  cost  over¬ 
run)]  =  3.20  -  0.98(MPT  word  percentage) 

The  third  significantly  predictive  model  suggested  that  the  percentage  of 
MPT  words  was  again  the  only  factor  that  made  a  unique  contribution  to 
the  prediction  of  GAO  program  unit  cost  increases.  Increased  percentage 
of  MPT  words  contributed  to  a  lower  likelihood  of  GAO  program  unit  cost 
increases.  This  model  resulted  in  the  final  equation  shown  here. 

logit[Pr(Y=GAO  program  unit  cost  increase)]  =  2.61  - 
1.05(MPT  word  percentage) 

The  fourth  significantly  predictive  model  suggested  that  ESOH  word  per¬ 
centage,  HFE/Hab/Surv  word  percentage,  and  MPT  word  percentage  all 
made  a  unique  contribution  to  the  prediction  of  Weapon  Book  amount  spent 
over  budget  cost  overruns.  An  increased  percentage  of  ESOH  words  or  MPT 
words  contributed  to  a  greater  likelihood  of  Weapon  Book  amount  spent 
over  budget  cost  overruns,  while  an  increased  percentage  of  HFE/Hab/ 
Surv  words  contributed  to  a  lower  likelihood  of  cost  overruns.  This  model 
resulted  in  the  final  equation  shown  here. 

logit[Pr(Y=Weapon  Book  amount  spent  over  budget 
cost  overrun)]  =  -0.61  +  5.23(ESOH  word  percentage) 

-  4.90(HFE/Hab/Surv  word  percentage)  +  1.06(MPT 
word  percentage) 


Cost  overruns  identified  by  Weapon  Books  are  more 
affected  by  the  presence  ofHSI-related  terminology 
than  are  total  program  cost  overruns  identified 
by  GAO  assessments,  program  unit  cost  overruns 
identified  by  GAO  assessments,  and  schedule 
breaches  identified  by  SARs. 

Analysis  of  the  hypothesis  with  all  predictor  variables  yielded  the 
Nagelkerke  R2  values  for  the  four  significant  models:  SAR  schedule  breach, 
R2  =  0.06;  GAO  total  program  over  budget,  R2  =  0.13;  GAO  program  unit  cost 
increase,  R2  =  0.12;  and  Weapon  Book  amount  spent  over  budget,  R2=  0.14. 
Therefore,  cost  overruns  identified  by  Weapon  Books  are  more  affected  by 
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the  presence  of  HSI-related  terminology  than  are  total  program  cost  over¬ 
runs  identified  by  GAO  assessments,  program  unit  cost  overruns  identified 
by  GAO  assessments,  and  schedule  breaches  identified  by  SARs.  However, 
the  Nagelkerke  R2  value  is  low  for  each  of  these  four  outcomes,  thus  render¬ 
ing  little  predictive  power. 


Model  Sensitivity  and  Specificity 

Sensitivity  and  specificity  were  examined  for  each  model  using  classifi¬ 
cation  plots.  Each  plot  describes  the  percentage  of  correct  classifications  for 
a  predictive  equation.  Four  models  indicated  a  significant  predictive  abil¬ 
ity.  Therefore,  the  four  models  were  examined  for  their  ability  to  correctly 
classify  cases.  Results  of  the  classification  tables  are  presented  in  Table  8. 


TABLE  8.  CLASSIFICATION  TABLES  FOR  EACH  BINARY  LOGISTIC 

REGRESSION 


Dependent 

Observed 

Predicted 

Total 

Percentage 

Overall 

Variable 

No 

Yes 

Documents 

Correct 

Percentage 

SAR  Cost 

No 

352 

0 

100% 

Breach 

Yes 

156 

0 

508 

0% 

69% 

SAR 

No 

500 

0 

100% 

Performance 

Breach 

Yes 

9 

0 

509 

0% 

98% 

SAR 

No 

349 

6 

98% 

Schedule 

Breach* 

Yes 

141 

5 

495 

3% 

71% 

GAO  Total 

No 

16 

78 

17% 

Program 
over  Budget* 

Yes 

10 

188 

292 

95% 

70% 

GAO 

No 

4 

53 

7% 

Program 

Unit  Cost 
Increase* 

Yes 

2 

215 

274 

99% 

80% 

Weapon 

Book 

Amount 

No 

180 

4 

98% 

Yes 

53 

11 

248 

17% 

77% 

Spent  over 
Budget* 


Note.  An  asterisk  (*)  indicates  significance  at  the  p  <  .10  level. 
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As  shown  in  Table  9,  sensitivity  and  specificity  were  also  examined  using  clas¬ 
sification  plots  for  each  of  the  16  MDAPs.  Data  were  separated  by  MDAP,  and  16 
models  were  examined.  Nine  of  the  16  models  indicated  a  significant  predictive 
ability  and  were  examined  for  their  ability  to  correctly  classify  cases. 


TABLE  9.  CLASSIFICATION  TABLES  FOR  EACH  MAJOR  DEFENSE 

ACQUISITION  PROGRAM 

MDAP 

Observed 

Predicted 

Total 

Percentage 

Overall 

No 

Yes 

Documents 

Correct 

Percentage 

DDG  1000 

No 

1 

6 

14% 

Zumwalt 

Class 

Destroyer 

Yes 

2 

28 

37 

93% 

78% 

EA-18G 

No 

10 

6 

63% 

Growler 

Aircraft 

Yes 

0 

19 

35 

100% 

83% 

F/A-18E/F 

No 

8 

1 

89% 

Super 

Hornet 

Yes 

1 

2 

12 

67% 

83% 

F-22 

No 

6 

14 

30% 

Raptor 

Advanced 

Tactical 

Fighter 

Aircraft* 

Yes 

0 

51 

71 

100% 

80% 

Ground/ 

No 

3 

0 

100% 

Air  Task 

Oriented 

Radar* 

Yes 

0 

3 

6 

100% 

100% 

Joint  Land 

No 

10 

0 

100% 

Attack 

Cruise 

Missile 

Defense 

(JLACMD) 

Elevated 

Netted 

Sensor 

System* 

Yes 

0 

6 

16 

100% 

100% 

F-35  Joint 

No 

1 

5 

17% 

Strike 

Fighter 

Aircraft 

Yes 

0 

52 

58 

100% 

91% 
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TABLE  9.  CLASSIFICATION  TABLES  FOR  EACH  MAJOR  DEFENSE 

ACQUISITION  PROGRAM  (CONTINUED) 

MDAP 

Observed 

Predicted 

Total 

Percentage 

Overall 

No 

Yes 

Documents 

Correct 

Percentage 

Joint 

No 

11 

5 

69% 

Tactical 

Radio 

System* 

Yes 

0 

29 

45 

100% 

89% 

Littoral 

No 

9 

4 

69% 

Combat 

Ship* 

Yes 

1 

13 

27 

93% 

82% 

Mine 

No 

11 

11 

50% 

Resistant 

Ambush 

Protected 

Vehicle 

Yes 

10 

16 

48 

62% 

56% 

P-8A 

No 

12 

2 

86% 

Poseidon 

Multi- 

Mission 

Maritime 

Aircraft 

Yes 

8 

0 

22 

0% 

55% 

Space 

No 

5 

4 

56% 

Based 

Infrared 

System 

High* 

Yes 

0 

39 

48 

100% 

92% 

V- 22 

No 

10 

6 

63% 

Osprey 

Joint 

Services 

Advanced 

Vertical* 

Yes 

5 

26 

47 

84% 

77% 

Vertical 

No 

3 

5 

38% 

Take¬ 
off  and 
Landing 
Tactical 
Unmanned 
Aerial 
Vehicle 

Yes 

4 

4 

16 

50% 

44% 

Warfighter 

No 

15 

11 

58% 

Information 

Network- 

Tactical* 

Yes 

6 

19 

51 

76% 

67% 
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Note.  An  asterisk  (*)  indicates  significance  at  the  p  <  .10  level.  An  additional  dependent 
variable  was  defined  from  the  six  dependent  variables.  If  there  was  any  breach  indicated 
by  any  of  the  dependent  variables,  then  a  value  of  1  was  assigned  to  this  new  variable.  If 
none  of  the  dependent  variables  indicated  any  breach,  then  a  value  of  0  was  assigned 
to  this  new  variable.  Because  the  Advanced  Deployable  System  (ADS)  had  no  breaches, 
as  indicated  by  the  six  dependent  variables,  the  new  variable  had  a  value  of  0  for  each 
case  (each  program  document).  Because  there  was  no  variance,  binary  logistic  regression 
analysis  could  not  be  applied  to  the  data  for  ADS.  There  were  seven  program  documents 
for  ADS,  and  those  seven  cases  have  been  omitted  from  this  table. 


Discussion 

In  reference  to  Hypothesis  One,  the  presence  of  HSI-related  words  in 
an  MDAP  document  may  be  associated  with  whether  or  not  a  program  will 
experience  a  schedule  breach  or  a  cost  overrun.  First,  assessing  individual 
predictors  from  three  significant  regression  models  suggests  that  a  schedule 
breach  identified  by  a  SAR  or  a  cost  overrun  identified  by  GAO  assessment 
of  total  program  cost  or  program  unit  cost  is  less  likely  to  occur  when  more 
MPT  words  are  present.  Second,  assessing  individual  predictors  from 
another  significant  regression  model  suggests  that  a  cost  overrun  identified 
from  Weapon  Book  budget  data  is  more  likely  to  occur  when  more  ESOH 
or  MPT  words  are  present.  However,  a  cost  overrun  is  less  likely  to  occur 
when  more  HFE/Hab/Surv  words  are  present.  Therefore,  considering 
SAR  data  and  GAO  assessment  data,  the  presence  of  terminology  about 
MPT  in  a  weapon  system’s  acquisition  program  documents  suggests  that 
the  program  might  not  experience  a  schedule  slippage  or  cost  overrun. 
Considering  Weapon  Book  budget  data,  the  presence  of  terminology  about 
MPT  in  a  weapon  system’s  acquisition  program  documents  suggests  that 
the  program  will  experience  a  cost  overrun.  Also  regarding  Weapon  Book 
budget  data,  the  presence  of  terminology  about  ESOH  in  a  weapon  system’s 
acquisition  program  documents  suggests  that  the  program  will  experience 
cost  overruns,  whereas  the  presence  of  terminology  about  HFE/Hab/Surv 
suggests  that  the  program  might  not  experience  a  cost  overrun. 

During  the  acquisition  life  cycle,  program  success  can  be  affected  by  various 
efforts  conducted  by  program  managers  and  systems  engineers.  As  shown  in 
Figure  2,  three  milestones  within  the  Analysis  of  Alternatives  (AoA)  occur 
prior  to  Milestone  A,  where  a  group  of  concepts  are  identified  and  compared 
among  one  another  (Department  of  Defense  [DoD],  2015).  Milestone  A  is 
when  the  Risk  Reduction  Decision  is  made,  whereby  a  specific  concept  is 
selected  for  further  development  and  resources  are  committed  to  the  mat¬ 
uration  of  relevant  technology  (p.  7).  Milestone  B  is  when  the  Development 
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Decision  is  made,  and  contracts  are  awarded  for  producing  and  testing  the 
concept  (p.  7).  Milestone  C  is  when  Low -Rate  Initial  Production  of  the  con¬ 
cept  begins  (p.  7). 


FIGURE  2.  ACQUISITION  LIFE  CYCLE 


Need  Identification 

(DoD:  Materiel  Development  Decision) 


Solution  Analysis 


Risk  Reduction  Decision 
.(DoD:  Milestone  A) 


Technology  Maturation  and 
Risk  Reduction 


Development  J 
Decisions  ' 


Requirements  Decision  Point 
(DoD:  CDD  Validation) 


^evelopment  RFP  Release 

Development  Contract  Award 
a  (DoD:  Milestone  B) 


Development 


LEGEND: 

A  =  Decision  Point 

CDD  =  Capability  Development  Document 

RFP  =  Request  For  Proposal 


Production 

Decisions 


Initial  Production  or  Fielding 
^DofcMilestoneQ _ 

Low-Rate  Initial  Production  or 
Limited  Deployment  and  Operational  Test 


k  Full-Rate  Production/Full  Deployment 


Figure  2  illustrates  the  sequence  of  decision  events  in  a  generic  program. 

It  is  not  intended  to  reflect  the  time  dedicated  to  associated  phase  activity. 


Production  Deployment, 
and  Sustainment 


Disposal 


Meanwhile,  a  program’s  cost  estimation  can  be  impacted  during  the  time 
period  leading  up  to  Milestone  B.  By  Milestone  B,  70  percent  of  a  system’s 
life-cycle  cost  will  have  been  determined  by  design  decisions  regarding  the 
program’s  features  and  efforts  (Deitz,  Eveleigh,  Holzer,  &  Sarkani,  2013; 
General  Accounting  Office,  1981;  Zimmerman,  Butler,  Gray,  &  Rosenberg, 
1984).  After  errors  have  been  made  at  the  Milestone  B  decision  point,  repair¬ 
ing  the  errors  or  compensating  for  them  costs  between  three  and  10  times 
more  than  the  cost  of  the  original,  erroneous  efforts  (Deitz  et  al.,  2013). 

Program  managers  and  systems  engineers  can  apply  the  observations  from 
this  investigative  study  to  their  understanding  of  human  considerations  and 
what  impact  human  considerations  have  on  the  development  of  a  given  pro¬ 
gram.  Systems  engineering  includes  HSI,  and  HSI  can  be  incorporated  into 
the  content  of  program  documents,  such  as  the  requirements  documents. 
Requirements  definition  is  one  facet  of  SE,  which  is  the  interdisciplinary 
approach  for  developing  systems  (INCOSE,  2012).  MDAPs  employ  SE  to 
conduct  weapon  systems  acquisition  for  the  DoD.  To  minimize  weapon  sys¬ 
tems  acquisition  costs,  the  DoD  created  the  Better  Buying  Power  mandate. 
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which  has  identified  some  focus  areas  with  appropriate  initiatives,  such  as 
(a)  eliminating  requirements  that  lead  to  nonvalue-added  processes,  (b) 
improving  how  requirements  are  defined,  and  (c)  inhibiting  requirements 
from  changing  over  time,  in  other  words,  requirements  creep  (DoD,  n.d.). 


Conclusions 

The  data  for  this  investigative  study  were  representative  of  different 
customers  within  the  U.S.  Government  (Air  Force,  Army,  DoD,  Marine 
Corps,  and  Navy)  and  of  different  types  of  weapon  systems  (aircraft,  com¬ 
munications  network,  ground  vehicle,  ship,  etc.),  which  helps  ensure  validity 
of  the  findings  among  customers  and  weapon  systems.  This  investigative 
study  looked  back  at  existing  MDAP  documents  in  a  retrospective  content 
analysis  as  a  means  to  look  forward  for  program  success.  Considering  how 
many  MDAPs  exist,  the  sample  size  was  relatively  small.  However,  the  value 
of  this  study  is  that  it  has  revealed  a  trend  that  HSI  practitioners  already 
suspected  and  that  can  be  examined  further  by  investigating  more  programs 
with  more  documents.  Exposing  trends  by  looking  at  historical  data,  such  as 
how  HSI  impacts  weapon  systems  acquisition,  is  informative  for  planning 
and  developing  future  systems. 
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